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The Wisconsin Learning-Centered Teacher Evaluation Study: 
Informing Policy and Practice 


Steven Kimball, Jessica Arrigoni, Herb Heneman, Elisabeth Geraghty, Daniel Marlin, 
Bradley Carl, Curtis Jones, Elizabeth Cain, Anthony Milanowski, and Katharine Rainey 


Major teacher evaluation changes have occurred across the nation during the past decade. 
Pressed by federal education initiatives and encouraged by research on promising teacher 
evaluation practices (Grissom & Youngs, 2016; Milanowski, Kimball, & Odden, 2005; Taylor & 
Tyler, 2012; Kane & Staiger, 2012), states and school districts have implemented evaluation 
systems using new measures of educator practice and student learning. 


A growing body of research has examined teacher evaluation policies and related impacts, 
including state implementation (e.g., Wayne et al., 2016; Brown et al., 2016), district 
implementation (e.g., Anderson et al., 2016; Malen et al., 2015; Putman et al., 2018), 
measurement qualities (Grissom & Youngs, 2016), performance feedback (e.g., Garet et al., 
2017; and Rowland et al., 2018), and linkages of teacher evaluation systems to other human 
capital management systems (e.g., Heneman & Milanowski, 2011). Relatively less 
implementation research focuses on the school level in a way that captures the multiple roles 
involved (i.e., principals, teachers, coaches, peers). 


Wisconsin adopted a system for educator evaluation that includes common measures, similar 
training requirements, and an emphasis on educator growth and development. School districts 
have considerable flexibility to adapt local practices and use results for local decisions. The 
evaluation of the Wisconsin Educator Effectiveness (EE) System has provided informative state, 
district and school reports on teacher and principal perceptions of the system. There is much to 
learn, however, about how schools are carrying out evaluation practices to support educator 
improvement. 


This report presents findings from the Learning-Centered Evaluation Study carried out during 
the 2017-2018 school year. We examine school-level educator effectiveness practices and outcomes 
within Wisconsin districts focusing evaluation efforts on educator improvement rather than 
accountability. The report includes four main sections. We first introduce the study by providing 
background on the EE system development in Wisconsin, an overview of the five principles of 
learning-centered evaluation, and the main study questions. In the second section, we summarize 
the study design. In the third section, we present findings on learning-centered practices in the 
schools, and observations on how evaluation affects teaching practice. The final summary 
section concludes with a policy response from the study sponsor and thoughts on future studies. 


I. Background on EE System 


In Winter 2010, the Wisconsin Department of Public Instruction (DPI) convened an Educator 
Effectiveness Design Team to develop a state model for teacher and leader evaluation. Following 
multiple planning meetings over the course of 10 months, the Design Team produced a report 
outlining a teacher and leader evaluation framework (Department of Public Instruction, 2011) 
intended to provide a foundation for the state evaluation system. According to the report, “The 
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primary purpose of the Wisconsin Framework for Educator Effectiveness is to support a system 
of continuous improvement of educator practice.” Following the report, DPI initiated work teams 
to develop the teacher and principal evaluation process, including measures of professional 
practice and student learning outcomes, and how those measures inform educator development 
and support. 


Subsequently, the Wisconsin Legislature passed Act 166 in 2012. The state legislation 
reinforced the DPI evaluation approach and required all districts to implement the new system by 
2014-15 using a state-developed model or equivalent model approved by DPI." The time allowed 
the DPI to create training opportunities and pilot test the new system, while leveraging an 
external evaluation to inform potential changes and supports. The system was pilot tested over 2 
years and fully implemented in the 2014-15 school year. Although a common evaluation system 
had potential to establish consistency in evaluation approaches, in keeping with local control, 
Wisconsin school districts have discretion in the use of evaluation for high stakes (e.g., employee 
retention, compensation) and low stakes (e.g., professional development) decisions. 


Figure 1 illustrates the main EE system processes as described in the DPI User Guide for 
Teachers, Teacher Supervisors, and Coaches (Department of Public Instruction, 2018). The 
typical evaluation cycle spans three years, which includes a Summary Year and 2 Supporting 
Years. The figure represents the basic steps during the Summary Year. Educators first receive 
orientation information at the beginning of the process and engage in reflection and goal setting 
to develop an Educator Effectiveness Plan (EEP). The EEP includes the Student Learning 
Objective (SLO) and the Professional Practice Goal (PPG). Educators meet with evaluators and 
peers during the year to discuss goal progress and observations. At the end of the year, educators 
with their evaluator or peers discuss results and plan for the following year. The main steps 
represent an annual improvement cycle with the emphasis on educator-developed goals, 
feedback from peers and evaluators, and three meetings that occur across the year to discuss 
progress and needed support. DPI stresses using the EE system for learning rather than 
accountability. 


1 The present study focuses on districts implementing the state model for teacher evaluation that centers on 
Danielson’s Framework for Teaching. Other districts use equivalence models, including the approach by CESA 6 
adapted from research by Stronge (2007; 2010). Both represent models of practice on which to center the evaluation 
process. 


Figure 1. Major Processes in the EE system Annual Cycle 


AUG-OCT 


*Orientation Meeting. 
Overview of the system 
measures and processes, 
identify who can provide 
support, discuss timelines 
and schedules. 


*Self-Review. Educator 
analyzes student, school, 
and personal data to 
determine areas of strength 


and areas for improvement. 


*Educator Effectiveness 
Plan. Educator creates an 
EEP, which includes an 
SLO and PPG. 


*Planning Session. Review 
EEP, discuss and adjust 
goals if necessary, identify 
evidence sources, actions, 
and resources needed. 
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NOV-APR 


*Evidence Collection & 
Ongoing Improvement. 
Centered on the educators 
goals, evidence is collected 
on artifacts (i.e., teaching 
materials, student work) 


and classroom observations. 


Written and verbal 
feedback occurs following 
observations. Evidence 
collection takes place 
throughout the evaluation 
cycle. 


*Mid-year Review. Review 
PPG and SLO, receive 
feedback and discuss 
progress. Goals may be 
adjusted if necessary. 


MAY-JUN 


*Process and outcome 
determinations. Includes 
assessment of degree of 
success in achieving SLO 
and PPG based on 
evidence. SLOs are self- 
scored by educators. 
Evaluator assigns a holistic 
SLO score in Summary 
Years based on SLOs 
across the cycle. 


*End-of-Cycle 
Conversation and 
Conference. Educators 
receive feedback on PPG 
and SLO achievement, 
discuss results on 
components of the 
professional practice rubric 
and SLO results. Identify 
growth areas for upcoming 
year. 


Drawing from pilot stage experiences as well as the literature on performance management 
and adult learning, DPI emphasizes learning-centered teacher and leader evaluation practices. As 
articulated in the teacher effectiveness guide (User Guide for Teachers, Teacher Supervisors, 
Coaches), there are five principles associated with learning-centered evaluation: (1) a context of 
trust that encourages risk-taking and learning from mistakes; (2) a common model of effective 
practice (i.e., the Framework for Teaching [FFT]) to center conversations about teaching and 
inform professional learning within and outside the evaluation context; (3) educator-developed 
goals that are regularly referenced to frame the evaluation process; (4) cycles of continuous 
improvement guided by specific and timely feedback to drive practice; and (5) integration of 
evaluation practices with other school and district improvement strategies. We briefly elaborate 


these principles below. 


Context of trust. Trust provides an essential foundation for student or adult learning (Bryk 
& Schneider, 2002). Trust within the evaluation context includes transparency in the 
requirements and criteria, trained evaluators to ensure consistency, encouragement of rigorous 
goals, and the understanding that educators will not be punished based on results. School 
administrators play key roles in building trust and encouraging risk-taking in the evaluation 


process. 
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Common model of effective practice. DPI selected the 2013 FFT by Charlotte Danielson as 
the basis for evaluating and supporting teaching practice. This framework has been the focus of a 
number of research studies demonstrating validity and reliability (see, e.g., Gates Foundation, 
2013; Taylor & Tyler, 2012; Milanowski et al., 2005). Although primarily used as a tool to 
support professional growth and evaluation, the Framework may also apply to mentoring, and 
other human capital management practices (Heneman & Milanowski, 2004). 


Educator-developed goals. To create greater relevance and promote acceptance, educators 
are encouraged to set goals based on analysis of student data as well as self-appraisal and 
reflection on their own practice. Goal setting can support professional learning and performance 
execution (see, e.g., Locke & Latham, 2013). Two educator goals center the Wisconsin 
evaluation and growth process: (a) the PPG and (b) the SLO. Together, these goals represent the 
annual EEP. Peers and evaluators are encouraged to support educator goals through dialog and 
feedback within a continuous improvement cycle. 


Continuous improvement guided by feedback. Improvement cycles represent intentional 
actions that entail goal-setting, evidence collection, reflection, feedback and revision. The 
Wisconsin evaluation system may facilitate continuous improvement through three main phases, 
including beginning-of-the-year goal setting, middle-of-the-year revision, and end-of-year 
reflections. Throughout the cycle, educators collect evidence of practice, receive feedback and 
support from peers and evaluators, and revise goals based on this information. 


Integration with school and district priorities. Integration of evaluation practices within 
larger school and district priorities can move what is typically an individual evaluative 
experience into an organizational improvement strategy. When classroom, grade-level, 
departmental, school, and district goals align, the evaluation process can be reinforcing and drive 
growth across the local education system. 


Each of the five principles of learning-centered evaluation has potential to enhance 
leadership practice, educator acceptance, and use of the evaluation system for improvement. 
When all principles are in place, we hypothesize that the system can contribute to improved 
individual and organizational effectiveness. The present study represents an initial attempt to 
address the presence and impact of growth-oriented school-based evaluation practices within 
districts embracing the EE System as an improvement strategy. Two primary questions frame the 
study: 


1. To what extent are schools implementing learning-centered teacher evaluation practices? 
2. How do local evaluation approaches that include learning-centered practices contribute to 
changes in teaching? 
II. Study Design 


At the request of the Educator Development and Support (EDS) office within the DPI, this 
report builds on the external evaluation of the EE system by providing a deeper look at school 
and district practices supporting educator growth and development. The study is primarily 
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qualitative, allowing for an extended exploration of how teachers, leaders, and instructional 
coaches are carrying out the evaluation process. 


In 2016-17, the study team initially reached out to districts thought to be carrying out growth- 
focused evaluation practices. Ten geographically diverse districts were included based on the 
results from the 2016 Wisconsin Educator, Development, Support, and Retention (WEDSR) 
Survey” and recommendations from educational leaders. The visits led to district-level case 
studies. We selected five districts from the initial 10 for a school-level study during the 2017-18 
school year. At that time, we also added schools from Milwaukee Public Schools (MPS) to 
provide insight into learning-centered practices within a large urban school setting. 


Within each district, we selected 2-3 schools, including schools at the elementary, middle and 
high school level when available. We reviewed WEDSR survey responses on perceived trust and 
feedback quality to identify schools where learning-centered practices were most likely 
occurring. Comparing schools based on teacher WEDSR survey responses helped narrow the 
selection when there was more than one school at a grade level. The process of selecting schools 
in Milwaukee was slightly different. Evaluation team members met with MPS staff to identify a 
list of schools believed to promote a learning-centered culture around EE and in which teacher 
responses on the WEDSR survey reflected this culture. Through this process, two schools agreed 
to participate, one K-8 school and one middle school with Grades 6-8. The final sample included 
16 schools.* Appendix A lists the districts and schools in the final sample and Appendix B 
further describes study methods. 


In each school, we interviewed teachers, evaluators (i.e., principals or assistant/associate 
principals) and coaches. Interview protocols are included in Appendix C. We collected 
documents, such as school improvement plans, and evaluation tools when available. We also 
asked teachers to complete logs following feedback interactions with their evaluators or coaches. 
For all schools except the Milwaukee schools‘, we collected SLOs from the prior academic year 
for a sample of teachers. 


This study aimed to examine EE implementation and results in schools and districts deeply 
engaged with the EE system. We selected a small sample of districts, schools within districts, 
and educators within schools to permit in-depth data collection within limited resources and to 
minimize study burden. We used purposeful sampling (Patton, 2008) to select the districts and 
schools rather than using random selection. Therefore, while the data provide rich descriptions of 
practice within the study schools and districts, the results may not be representative of other 
schools and districts. 


2 www.uwm.edu/wsedsr 
3 One district, Cashton, is comprised of only two schools, an elementary school and a combined middle/high school. 
4 SLOs were not part of the data requested from the MPS schools. 
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III. Findings 


The study design allowed for in-depth examination of how teachers, principals and instructional 
coaches described their experiences with local evaluation practices. We next present findings on 
the main study areas of (1) school evaluation practices that reflect the learning-centered 
principles, and (2) how learning-centered evaluation contributes to changes in teaching practice. 


How Local Evaluation Practices Reflect Learning-Centered Principles 


Context of trust. Leaders in the study districts and schools emphasized an environment of 
trust within and outside of the EE process. Building trust has been an intentional process of 
getting staff comfortable with the change in policy that includes greater focus on instructional 
practice. Each district communicated early about the growth orientation of the evaluation system, 
made available training and resources to address questions and support the process, and set the 
tone for formative uses. 


The study districts conducted extensive district-wide training at the initial system 
implementation stages. Much of the training focused on system structure, the FFT, and the 
technology platform. Most of the initial training occurred within the districts, but some districts 
also sought trainings from their local Cooperative Educational Service Agency (CESA). With a 
basic knowledge base developed, training moved to other aspects of the system—for example, 
components within the Framework, artifact collection, and technology. Each district and 
participating school includes some level of basic EE System training for beginning educators, 
including guidance on how to develop SLOs. In most districts, once trained in the process, 
principals trained their staff. 


The DPI requires evaluators to participate in ongoing calibration training on the FFT, which 
occurs twice a year. In addition to the calibration training, several districts hold team calibration 
walks for evaluators in order to build consistency between schools. Evaluators also meet 
regularly to discuss the evaluation process, concerns raised by teachers, district evaluation 
trends, and how to improve instructional feedback. In Wausau School District, evaluators 
attended in-depth training on teaching skills and related feedback based on the book, The Skillful 
Teacher, by Jon Saphier and colleagues (2008). In Baraboo School District, district EE leads 
provided sample feedback to evaluators, and as a group, they reviewed and discussed the types 
of feedback and focus that they wanted as a district. In another district, training for evaluators 
concentrates on creating non-judgmental observation statements and action-oriented feedback. 


Across the schools, administrators were available, responsive, and transparent about the 
evaluation process. As a teacher explained, “our administrators are visible, that helps build 
trust.” Evaluators in each school in another district described avoiding excessive note taking 
during observations or post-observation discussions to create an inviting process and build 
relationships. Evaluators sought to provide formative feedback rather than summative 
assessments of teaching practice. They also tried to visit classrooms outside of the formal 
evaluation process to maintain visibility and get both teachers and students comfortable with 
their presence. An assistant principal emphasized the importance of structuring formative 
conversations: “The conversations have to be coaching conversations. If they’re feeling attacked, 
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or that you’re always pointing out the negative, you’re never going to have the same trust.” 
Further, in most of the schools, teachers expressed that their professionalism is respected; as 
reflective of other comments, one teacher mentioned that administrators “honor and believe in 
and value education and our opinions as teachers.” 


An essential component of trust across the schools stems from the fact that the evaluation 
process is not punitive. Rather, administrators conveyed, and teachers affirmed, the process is 
one of professional growth that permits risk-taking in furtherance of that goal. Teachers do not 
fear reprisal if they are unable to meet their goals, as one teacher said: “Nobody here feels a 
stigma attached with not doing well at something.... [I]t’s much more of a learning culture... 
[W]e do a really good job of making it about the kids.” Teachers in several districts indicated 
that they are able to modify their goals mid-year if they are too challenging, as one said: “If 
[teachers] get to mid-year, look at the goal and realize it’s too challenging, they can change their 
goal. The administration does a good job reinforcing the idea that they would rather have 
teachers set high goals and work to achieve them.” 


Collaboration can also foster trust. In this context, trust-building collaboration occurred 
between evaluators and teachers and between teachers themselves. One principal stated, “They 
[teachers and evaluators] don’t always agree but it is a good conversation regarding evidence as 
to how they got there.” In multiple districts, teachers are encouraged to collaborate in 
professional learning communities (PLCs) or observe their peers during class time, and 
administrators will offer to fill in for teachers so that they are able to do so. 


Use of a common framework. Schools within districts adopting the state model for 
evaluation use the FFT as the foundation for evaluation goal setting, discussions, and 
performance determinations. Educators and leaders across the study schools spoke positively of 
the Framework as a model of professional practice. The extent to which the FFT informed 
school-based induction, mentoring, coaching, and professional development varied. 


Each school utilized the FFT as a common framework of effective teaching practice. One 
principal stated that his school integrates the FFT “completely, [by building] our conversations at 
a leadership level around that, we’ve done calibration around that, and then compared and had 
conversations between us about scoring a teacher in those areas.” As another administrator said it 
provides “fences for conversation around instruction.” A coach also mentioned referencing the 
FFT when conducting class visits and providing formative feedback. According to an assistant 
principal, “I would struggle to provide meaningful feedback without the Framework.” Further, 
several of the districts used the FFT to identify district priorities, which ranged from practices 
such as questioning and discussion, checking for understanding and communicating learning 
targets, to broader district initiatives such as personalized or project-based learning. Many 
schools also encouraged the use of the FFT in the development of SLOs and PPGs. 
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The Cashton School District embraces the common model of practice represented by the 
Framework for Teaching. In addition to forming the foundation for evaluation discussions and 
goal setting, district and school leaders make connections between the FFT and the district’s 


approach to PLC meetings and Positive Behaviors Interventions and Supports implementation. 
These connections help teachers and leaders create coherence around improvement efforts. 


The extent to which the FFT is explicitly referenced throughout the districts varied. At one 
school, a teacher indicated that “we talk a lot about the different domains,” while a teacher in 
different school said, “it’s not like we walk around and reference Danielson,” and another said 
that the FFT is “almost second nature.” Respondents from a different district indicated that the 
FFT is “always there” and “ingrained in conversations,” but may not be systematically 
referenced during conversations. This variation may result from increased familiarity with the 
FFT over time, with conversations more “behind the scenes” now than they were in the system’s 
early years. Interviewees in multiple districts mentioned that outside of the evaluation process, 
the FFT tends to be largely utilized through support provided by school leaders and coaches to 
new teachers. 


Educator-developed goals. To explore the extent to which educators developed their own 
goals and the quality of the goals, we asked teachers and leaders to describe the goal setting 
process, use of goals to inform practice and evaluation discussions. We also explored the 
qualities of the goals and related plans by collecting and analyzing a sample of SLOs and PPGs. 


Although most educators had flexibility in setting goals based on the classroom data and 
personal reflections on practice, school and district leaders encouraged goal alignment between 
individual, school and district priorities. Educators were allowed to support one another with 
goal development and many collaborated in doing so. School and district leaders said that they 
encouraged educators to pursue “stretch” goals, but a separate analysis of SLOs suggests that the 
development of rigorous, hard to reach goals was not common. 


Teachers across the study described a similar process of developing SLOs aligned to school 
and district instructional priorities and PPGs based on self-reflection. In most cases, when 
developing SLOs, the district conveyed district-wide priorities to staff early in the year, either as 
part of a staff professional development day or through other communications. One district 
compiled a large slide deck containing summaries of assessments to share priority areas. Another 
created a “heat map,” or data matrix with values represented with colors, to identify areas of 
concern or strength related to learning, attendance, and behavior goals. Teachers in most districts 
were strongly encouraged to set goals that align with district and school priorities, and most 
teachers did so without being required. The aligned goals frequently included teachers’ 
classroom- or grade-level data. 


The rest of the SLO process typically unfolded similarly. Teachers often developed SLOs 
with grade level or departmental teams. Nearly all of the districts in the study had schools in 
which setting tiered SLOs is either in practice or encouraged, in an effort to be more inclusive 
and help focus on student sub-groups. Tiered SLOs help educators identify a goal and set targets 
for groups of students so that all students are represented. While evaluators and coaches were 
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available to work with teachers on developing their SLOs and identifying strategies, most 
requests were limited to questions about the electronic platform for uploading goal statements 
and evidence. Teachers also reported little use of the DPI SLO quality criteria and SLO rubric to 
guide goal development. For example, one teacher mentioned that they knew about the quality 
criteria and rubric, but felt that teachers newer to the profession use these tools more regularly. 


As part of the Learning-Centered Study, we closely examined SLOs from the participating 
schools.’ We collected and reviewed 81 SLOs completed during the 2016-17 academic year on 
the following dimensions: SLO completion, the focus of student learning goals, instructional 
emphasis, and goal qualities using the DPI SLO Quality Indicator Checklist. The analysis found 
that most SLO elements were completed, but quality varied. Based on the DPI SLO Quality 
Indicators, we found quality highest in alignment and evidence sources, and lowest in targeted 
growth and baseline data. When breaking out SLO quality ratings according to where teachers 
were in the evaluation cycle (Summary Year vs. Supporting Year), we found that Summary year 
teachers’ SLOs were of higher quality for each of the Quality Indicators, although differences 
between Summary and Supporting teachers’ SLOs were larger for some of the Quality Indicators 
(such as Alignment) and smaller for others (such as Interval). Regarding content focus, nearly 
half of the SLOs were in English Language Arts, reading, or writing. The remaining SLOs were 
in science, math, physical education, and Social Studies. Almost all educators reported plans to 
collaborate on their SLOs. Most stated they would collaborate with their evaluator, while the 
other educators stated they would collaborate with a coach or a peer. 


In contrast with our SLO analysis, many of the teachers in the study reported that they are 
comfortable setting stretch goals that include all of their students. Evaluators for the most part 
also felt that teachers were setting rigorous goals. Evaluators reported providing feedback to 
teachers during the planning sessions on how to set more rigorous goals if the goals were not 
challenging. For example, an evaluator in an elementary school had an all-staff meeting to 
discuss how teachers could establish goals that were more rigorous and discussed how each 
teacher should be setting tiered goals. Goals perceived to be less rigorous were described as 
focusing on students who are already close to proficient, or including only a subset of the class. 


PPG development tended to be an individualized process with teachers identifying their own 
practice areas and setting their own goals. Areas of focus typically emerged from the teachers’ 
self-review using the FFT or other self-identified areas of interest. Few PPGs aligned with the 
teachers’ SLOs. Additionally, PPGs were sometimes described in very general terms (e.g., “learn 
more about computer science and apply learning in the classroom”) or quite specifically (e.g., 
“students will utilize differentiated sight word games and the use of flash cards with the folding 
in strategy at least 2 times a week”). Some teachers carried over their PPG from one year to the 
next, with modifications following end-of-year reflections. In a couple of instances, teachers did 
not have time to implement their PPG as intended and perceived it as a lower priority than other 
teaching practices. 


°The SLO study is summarized in a separate report. 
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Continuous improvement guided by feedback. Across the year, we asked teachers about 
how they received and used evaluation feedback as well as collegial and coaching conversations. 
Additionally, we asked teachers to complete feedback logs following conversations with 
evaluators or coaches. Teachers reported a number of sources from which they received feedback 
and how they used feedback to inform practice and revise goals. 


Goal-specific feedback and feedback related to the components of the FFT and district 
instructional priorities tended to come early in the school year, and primarily from evaluators and 
coaches. This feedback was usually shared as part of a meeting with a coach or evaluator, or 
documented as part of the EE process in an email or other electronic format. 


Most teachers described feedback during the goal setting stage or revision process that 
helped them make some changes to their goal statements or led them to think of different 
strategies to goal implementation. Teachers also described observation-related feedback that 
helped them refine practice. When describing feedback from her evaluator, one novice teacher 
said, “[T]he biggest way evaluations help is through reflection and feedback. That’s the biggest 
part I get out of it. As an initial educator, I look for feedback to stay on track and improve. That’s 
the biggest encouragement. So they can see how I improved and used feedback and adjusted.” 


In Baraboo School District, Educator Effectiveness leads gather written mini-observation 
feedback from all of the evaluators, black out names for confidentiality, and circulate the 
feedback statements to all evaluators. The group, along with the district EE leads, then critiques 
the feedback individually and discusses the types and areas of feedback that as a district they 


want to provide. Evaluators reported that it was helpful to see what feedback others were 
providing and discuss the results as a group. One evaluator shared that it “completely changed 
how I wrote my feedback, [I] would love to continue to do that.” 


Teachers and evaluators typically viewed feedback as being given in the main evaluation 
milestones of goal setting, post observation discussions, mid-year review meetings, and end-of- 
year conversations that lead to possible insights for the following year. It was rare to hear about 
fine-grained approaches to continuous improvement, with feedback and action steps taking place 
in a shorter period. This finding reinforces evidence from the feedback logs completed by 
participating teachers. According to the feedback log analysis, few teachers report specific next 
steps based on their feedback within a determined period. 


In addition to seeking support from evaluators, teachers reported relying on peers, grade- 
level/content-area teams, and PLCs for support. Teachers commonly reported that the feedback 
from peers was highly valuable and, in some cases, more readily available than evaluator 
feedback. One teacher shared, “I appreciate the feedback from my administrator, but that is only 
twice a year. I meet with my team once a week, that’s where I think a lot of my growth is 
happening.” 


Coaching activities. Coaching structures vary across the districts, and at times, across 
schools within a district. In the study districts, coaching was categorized as (a) EE 
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implementation support or (b) coaching on instructional strategies and feedback on professional 
practice. 


Educator effectiveness implementation coaching was typically the function of EE coaches, 
usually classroom teachers who had the additional role of providing assistance with the online 
evaluation system, initial goal development, and artifact collection. EE coaches typically 
provided support during professional development days or outside of the regular school day. 
Teachers frequently initiated contact with the EE coaches. In two districts, the EE coach role was 
limited to supporting basic EE implementation tasks. In these districts, coaches and teachers 
expressed the desire for the coaching to include direct interactions on classroom practice. With 
each year of system implementation, EE coaches were called upon less for technical support on 
the system. Teachers newer to the profession were more likely to seek such support. 


Franklin invests in coaching as a districtwide professional improvement support where 
“everyone is coached.” District leaders coach principals and one another. Thirteen district- 
based system specialists provide coaching to teachers in several district priority areas: math, 
literacy, personalized learning and technology. The math specialists are assigned to a school 
while the literacy, personalized-learning, and technology specialists work across schools. 
Specialists apply a gradual release coaching model. Following an initial visit to learn what 
supports teachers would like, the specialists create a plan with the teacher, then model practice 
” “we do,” “you do” approach. Specialists check back with teachers later to see 
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using an “I do, 
how well practices have taken hold and to discuss transfer to other instructional practices as 
well as stretch activities to continue growth. System specialists do not play a formal role in the 
EE process, but they complete training on the Danielson FFT. Each coach takes training to be 
certified in the FFT as the Framework served as the foundation for dialog about and planning 
for instruction. They felt the process was valuable and powerful. As one specialist said, “[I]t 
made me rethink everything.” 


Content experts (i.e., reading specialist, literacy coach, math coach) who were school- or 
district-based typically provided instructional coaching. Instructional coaches often supported 
teachers on an area of practice identified by the teacher, which may or may not have included 
their SLOs or PPGs. In one district, teachers were encouraged to work with school-based 
instructional coaches to help refine goals, develop strategies supporting their goals, and for 
ongoing goal monitoring. In two districts, the instructional coaches conducted observations and 
provided teachers with feedback. In one particular school in a different district, coaches and the 
leadership team identified a component of the FFT to focus on monthly; the team, including the 
coach, then conducted non-evaluative walkthroughs focused on that component twice during the 
month. Coaches also supported teams of teachers during PLC or other grade-level/content-based 
collaborative work time. As described above, in many schools, instructional coaches facilitated 
professional development activities. 


In short, getting feedback and reflecting on practice throughout the school year is a formal 
part of the EE process through milestone meetings between educators and evaluators and through 
educator self-assessment. Feedback also informally occurs through peer collaboration, EE goal 
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alignment and school-wide instructional priorities tied to the FFT. Feedback on goals and 
practice was intended to generate change or affirm practice and inform improvement; however, 
given limited evidence of planned next steps and follow up, feedback does not fully represent a 
continuous improvement process. Instructional coaches also provided educators with feedback 
and, in some cases, those coaching relationships also allowed for continuous improvement 
discussions. 


Integration with school and district priorities. To learn about the potential integration of 
the evaluation process with school and district priorities, we asked school administrators, 
teachers and coaches to share their school’s main instructional priorities and describe whether 
and how the evaluation process helps them address those priorities. Instructional priorities 
ranged between content areas like literacy and mathematics to instructional practices like 
questioning and discussion techniques and assessment and grading. Some administrators 
described priorities as they relate to the FFT. More often, school priorities were translated into 
School Learning Objectives and Student Learning Objectives. The study districts also integrated 
the evaluation process with PLC time. 


One administrator stated that they work to make sure there are “clear connections between 
priorities, practice, etc., making sure that teachers see that everything is connected.” Another 
administrator said that collaboration around the evaluation process helps build staff capacity: 


My mantra is that I don’t need to hire mentors for teachers because everyone on 
campus should be a mentor. It almost sounds like it’s too good to be true, but EE is at 
the center of our collaboration opportunities, and it’s directly connected to our 
conversations about professional growth. 


Another administrator stated that the building improvement goal and School Learning 
Objective are the same, that they talk about the “action steps that teachers generate around school 
improvement goals” in terms of their own SLOs, and that “there is a direct link between 
evaluation process and school priorities.” At another school, the administrator said that school 
improvement plans are “always tied to the Framework for Teaching” and “tied to the observation 
and feedback connected to the EE process.” Similarly, an administrator shared that the 
instructional priorities “help ... gauge what we are looking for in the classroom.” 


In several districts, school level SLOs were developed and then teachers, either individually 
or in teams, developed aligned SLOs using their own classroom data. A teacher in one of the 
districts with such a process stated, “[T]he evaluation process supports the instructional 
priorities.... [OJur administration likes to say we are working smarter, not harder. It does make a 
lot of sense, we are all focused on the same things so our discussions can be really high level 
around the same topics.” 
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Kettle Moraine implemented a district-wide initiative on personalized student learning that 
grounds their school and district priorities and approach to EE. The district uses the EE process, 
including goal development, observations, and the FFT, to help teachers receive feedback in 
relation to the district’s personalized learning approach. The process is reinforced by the 


application of teacher developed and district approved micro-credentials. As explained by a 
district leader, through the use of micro-credentials, PPGs and SLOs, they are personalizing 
learning for teachers while meeting district instructional goals. 


Professional development purposefully aligned with school and district priorities, SLOs, and 
the FFT is another way districts work to integrate EE. Several districts used the FFT as a guide 
for professional development by, for example, connecting specific instructional strategies to 
practices, breaking down the components of the FFT so that educators further understood the 
Framework, and demonstrating what practice looked like at each of the FFT levels and what 
evaluators were looking for when observing. 


In one district, EE leads reviewed observation feedback and practice ratings to identify 
professional development needs. Related professional development was then offered during 
district-wide PLC time. In each of the districts, teachers had dedicated PLC time to work on EE 
activities and attend professional development. Respondents also referenced professional 
development that supports SLOs, which occurred during district-wide, school-wide, or team- 
based activities. SLO-related PD was typically led by school-based or district-based coaches. 
Teachers also reported that they used feedback from the EE system to help them identify their 
own professional development. In one district, teachers used EE feedback to inform their micro- 
credential selection. 


How Evaluation Contributes to Changes in Teacher Practice 


To address the question of how the local evaluation practice contributes to changes in teacher 
practice, we examined interview notes from discussions with teachers, coaches, and evaluators in 
the participating schools, teacher feedback logs, and feedback log survey results. We found that 
learning-centered practices across schools help: (a) promote sustained instructional focus, (b) 
provide a common instructional language, (c) contribute to feedback from evaluators, and (d) 
support collaboration among peers. 


Promotes sustained instructional focus. Teachers commonly reported that the evaluation 
process encouraged them to sharpen and maintain their focus on student achievement goals and 
instructional practice. Although in some cases teachers indicated their practice may have 
included a focus on similar student learning goals regardless of the evaluation process, the SLOs 
and PPGs formalized their goal setting and reminded them to check in and be more systematic. It 
helped them make explicit connections between monitoring student progress and their own 
instructional practices, particularly practices related to the SLO. As one teacher explained, “the 
biggest thing [is that it] gives you something to focus on. So many things come at you all the 
time, but the SLO and PPG allows me to narrow my focus and helps keep me from being 
distracted.” 
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Common instructional language. In addition to the evaluation process encouraging 
reflection, providing regular checkpoints, and allowing opportunities for conversations among 
peers and between teachers and evaluators, the FFT was described as providing a common 
foundation for instructional dialog and goal setting. As a teacher explained, “[The FFT] helps me 
think about having balance in the classroom and to identify strengths and areas to improve. I 
used it in college so had knowledge.” Another elaborated on her use of the FFT with her team 
teaching colleague: 


I can think about it when [my team teacher] and I do self-reviews together as part of 
our collaboration. I would say we are pretty close in what we do for our job. When we 
did it this year, we saw other things to add. We talk about it in the Framework. And 
when we talk about our PPG, what to improve on, when we talk about notes home, 
goal setting, parent conferences, that constantly comes into play because that’s the 
piece we are working on within the Framework. 


The common language of the FFT also helped support feedback during interactions between 
teachers, their peers and evaluators. 


Feedback, interactions with evaluators. Most teachers were receptive to feedback and 
suggestions for changes to practice based on observations or goal-related discussions during 
check-ins at the beginning, middle or end of the year. Although few teachers described dramatic 
changes to practice due to evaluator feedback, and some mentioned no change, most described 
examples of feedback that provided ideas for possible changes as well as encouraging reflection. 
The following descriptions provide examples of evaluator feedback as described by teachers. 


The biggest thing that we have focused on ... was questioning, not that he would 
criticize my questioning, there was a lot of good, positive feedback into the questions 
that I do use, but he does a fabulous job in getting me to think more about, ‘Okay, how 
could you have reworded that question to make it even more open-ended for that 
student,’ and that’s what I’m constantly trying to improve on, to get the kids to do the 
thinking. And not just be the answer. 


And another teacher described how she applied the observation feedback: 


In beginning of the year, one thing related to how to get students more engaged in 
leading activities or showing ownership of the activities. So, I did make that change 
and incorporated into morning routines. I also got feedback on pacing of the lesson 
during math—I think it was after the long observation, when I went around and gave 
feedback to small groups, but he noted that I could provide the same feedback to 
others, even the students who could complete the tasks independently... I was touching 
base with those students but not the students who were doing well. [The principal] 
thought providing feedback to all students would be valuable. He wanted me to give 
feedback to students on task as well as others so everyone gets a personal touch. 


Through their feedback log responses, teachers also noted feedback and related 
recommendations they received and how they would act on the feedback. Table 1 includes 
examples of feedback described in the log entries. 
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Table 1. Feedback Log Responses 


What, if any, were 
the recommended steps to improve? 


I was recommended to change my SLO to 
make it more rigorous... [It] wasn’t rigorous 
enough after it was written because it didn’t 
allow for students to be at proficiency, but 
rather just gain one point on the district rubric. 


Do higher questioning with my guided 
reading group. Increase my wait time when 
asking a question. 


One area that I also asked my evaluator to 
look for was my wait time. That is an area 
that I have always had a difficult time with. 
She had several recommendations to help me 
increase my wait time. Additionally, she 
noted that although my wait time is quick, the 
students remain actively engaged because of 
the quick turn-around time. 


In what ways might you monitor student 
understanding and progress toward the related 
NGSS evidence statements in the project 
during the work time? How might students 
self-assess their understanding? How might 
you structure the closure of the lesson for 
students to consolidate their understanding 
and collectively share what was learned 
during the class? 


We discussed the importance of hands on 
activities and trying to get students to 
understand misconceptions in science so that 
understanding can result. We discussed how 
this could be done by having smaller groups 
to enhance one-to-one learning. Also, 
continuing conferencing with all students to 
ensure learning. 


15 


How, if at all, will you apply 
the feedback and recommendations? 


I made changes to my SLO as were suggested. 
My team and I continue to problem solve 
ways to help the students that we serve this 
year catch up to their grade level peers. 


I will look into having the students do more 
discussing during guided reading instead of me 
doing all the work. I would like to provide them 
with jobs to do this. I will approach my literacy 
coach to help me find resources to do this. 


I have already taken steps to increase wait 
time. I continue to look for ways to increase 
student discussion. 


I reflected on the feedback and am looking at 
incorporating it in future lessons. 


I will for sure apply all that we discussed. For 
instance, I will look not only at the ACT 
scores but reflect on classroom learning. 


Learning-Centered Teacher Evaluation 


Another teacher spoke of multiple aspects of the system, including feedback, supporting 
individual practice, and school goals: 


The evaluation process is driven by feedback, how that addresses our strategic plan 
and the achievement gap. We focus on the students and student outcomes. When we 
set an SLO we make sure it is data-driven; when we check it, we review data and 
when we received feedback and we are encouraged to collaborate with other 
professionals [we are also] addressing our strategic plan, maximizing the potential of 
all learners. 


Similarly, an EE coach also explained that evaluation practices, including evaluator 
feedback, aligned with building goals to impact instructional practice: “I think a combination of 
the evaluation practices and the building goals have driven some new student engagement 
strategies ... those two things and administrators being able to give feedback on engagement.” 


Teachers were also asked to rate the quality of feedback received when they completed their 
logs after feedback interactions. In an analysis reported separately, teachers rated feedback 
quality highly for accuracy, specificity, and timeliness and substantially lower for actionability 
(i.e., expedient use), whether it included strengths and weakness or next steps toward improving 
performance. These last findings could help explain some of the moderate perceived impact of 
the evaluation process on teaching practice reported by teachers and also suggests limitations 
with the use of the local EE processes for continuous improvement. 


Collaboration with peers and coaches. Teachers commonly discussed the positive impact 
of the collaboration facilitated by the EE process on goal setting, data review, and feedback from 
colleagues. Teachers frequently collaborated with colleagues during grade-level, cross-grade or 
departmental PLC meetings. During these meetings, they reflected on practice, and shared data, 
strategies and resources. One teacher shared an example of the collaborative aspect of the EE 
process when describing work with peers on a common SLO: 


So, during PLC time we have come together as a department on growth [with our 
SLOs] and whether we were seeing it or not seeing it. We tried strategies about what 
was working or not.... [It was about] repetitive practice with vocabulary and 
deliberate focus on topics that showed growth but depended on passing the ACT and 
question types. We were able to single out conflicting viewpoints, and thought about 
how to get at that, like the main idea of researcher and comparing and contrasting. 


In another school, a teacher shared that the school administration, “[encourages] us to 
collaborate with each other around our goals. That’s really significant. Teachers work so much in 
isolation, it’s great to work together, to get ideas and have a discussion. It’s a very collaborative 
building. That’s the most powerful part of the EE process.” 


School administrators encouraged collaboration and created opportunities for teachers to 
share their work on their SLOs and PPGs. Collaboration and feedback was also evident through 
discussions of coaching interactions. In two of the districts, coaches had a more hands-on role 
providing direct feedback with teachers. In these districts, there was a greater opportunity to 
collaborate with coaches. One teacher explained that she had more collaboration with the math 
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and literacy coach than her evaluator, sharing the following example: “[I] just had the literacy 
specialist in, which was an optional sign up, but was something she puts out as an option and | 
wanted to see her to be ready for the fall. So, I take opportunities to do it, but couldn’t say there 
is a lot of collaboration with [my principal].” 


IV. Summary 


This study closely examined school-level practices with the intent of informing district and 
state efforts to enhance the potential positive impact of teacher evaluation practice. 


Evaluation practices in the study schools contributed to a sustained focus, provided a 
common instructional language, encouraged a cycle of feedback and revisions, and enhanced 
peer collaboration. The evaluation process has also built upon and contributed to a climate of 
trust. Largely, teachers were encouraged and willing to embrace a common framework for 
teaching, set rigorous goals, collaborate with peers, and incorporate feedback into practice 
through the evaluation process. The evaluation approaches in the participating schools have gone 
beyond basic compliance, with each school trying to emphasize ongoing professional growth and 
development. In addition, the districts have adapted the evaluation process to meet local 
priorities and context, thereby increasing the potential to positively impact individual, school and 
district performance. 


Even with the positive overall findings, each school can improve its processes. 
Implementation appeared most meaningful in schools where the five principles were present, but 
schools (and even districts as a whole) varied on the extent to which this was the case. For 
instance, most schools could strengthen the use of the evaluation process as a cycle of continuous 
improvement, with additional training and time required to generate the feedback required for 
rapid improvement cycles. Better communication about the intent of artifacts and examples of 
high-leverage evidence sets could help educators narrow in on evidence that is not only 
streamlined to collect, but that also has value to professional practice. Moreover, district 
priorities outside of EE, while integrated with EE to some extent, in some cases tended to take 
priority over the purposeful implementation of the system. 


Recommendations for Enhancing Learning-Centered Evaluation with a Response and 
Policy Perspective by EDS Director 


This section provides suggestions for DPI, other state education stakeholders, and district and 
school leaders to build on successes and strengthen learning-centered teacher evaluation. Based 
on the study findings as well as discussions with district leaders during briefings on the school 
and district reports, these recommendations relate to state level communication and engagement, 
professional learning opportunities, and local process improvements. A response from the DPI 
EDS Director, Katharine Rainey, follows the recommendations. 


State level communication and engagement 


e Provide topic-specific briefs and links to resources on EE implementation and rapid cycles of 
continuous improvement for districts, schools, and other organizations. These briefs could 
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include stories or vignettes demonstrating effective practices, followed by selected tips and 
tools for additional information. 


e Create district guidance on how the EE process and products can facilitate the educator 
career trajectory from initial educator to master educator and National Board Certification. 


e Provide guidance to teacher and leader preparation programs on how they can better set up 
pre-service teachers and educational administrators for success with the EE system. This 
includes, but is not limited to, training in all aspects of developing and implementing 
effective SLOs and PPGs, and coaching and feedback. 


e Engage professional associations to consistently message about opportunities to enhance 
learning-centered evaluation. 


e Illustrate how districts can align human resource management systems with the research- 
based framework (i.e., FFT or equivalent) at the center. 


e Hold an annual EE conference or regional events to highlight learning-centered practices 
across districts. 


Professional learning opportunities 


e Create online learning opportunities for initial educators. This can supplement orientation 
materials and help build on district induction strategies. 


e Enhance current and future statewide professional learning offerings in collaboration with 
other state education stakeholders, such as professional associations and the CESA Network, 
to build educator and leader understanding of the EE system and ways in which they can 
leverage the system to improve learning at the individual, school and district levels. The 
Leading for Learning series is one such opportunity to create consistent messaging and 
jointly develop shared resources. 


e Develop in-person and online SLO, coaching, continuous improvement and feedback 
learning tools. These opportunities would be especially valuable to districts with limited 
capacity to provide or develop within-district training. CESAs are well positioned to support 
districts with these tools. 


e Educators would benefit from training and related resources on high-leverage evidence sets. 
Additionally, new as well as more experienced educators could benefit from ongoing training 
on the components of the FFT. Artifact training can be incorporated in orientation trainings, 
reiterated in meetings throughout the year with evaluators, and supported by coaches. In 
many cases, current school and district professional development can be aligned to the FFT. 
Drawing connections for teachers will help reinforce the use of the FFT as the common 
language related to instructional practice. 


e SLO training related to developing rigorous SLOs based on classroom level data, developing 
tiered SLOs that address equity, SLO monitoring, and SLO assessment that utilizes the SLO 
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Quality Indicator Checklist and SLO Scoring Rubric. Districts can utilize DPI’s updated SLO 
training: “Writing Quality Student/School Learning Objectives (SLO).” 


Local process improvements 


Create dedicated time for teachers to complete EE activities successfully with support from 
peers and coaches. Provide time for peers to collaborate on goal planning, monitoring, 
refining related instructional strategies, and assessment development. Providing educators 
with this time can set educators up to succeed, convey that leaders place value on and 
importance in the system, and help build trust in system uses. 


Provide district-specific calibration training and feedback training for evaluators. Including 
calibration training beyond the required minimums builds evaluators’ confidence and skill in 
conducting evaluations and providing feedback that in turn supports educator confidence and 
trust in the system. 


Take advantage of implementation flexibility to experiment with different approaches to 
observation cycles, such as substituting longer, formal observations with more frequent, 
shorter observations. These could be structured as a short-cycle improvement process with 
evaluators or peers identifying practice goals with teachers, observing that practice in 
frequent, unannounced visits, setting small goals based on observation-based dialog, then 
following up to see how the practice worked or whether it needs refinement. 


Provide instructional coaching that reinforces school and district improvement goals, as well 
as individual and school EE goals, and is available to teachers during EE time, PLC time, 
planning periods and classroom visits. Allow coaches to observe teachers and provide 
formative feedback related to EE goals and practice. 


Related to the above point, coaches need regular training to enhance skills related to goal 
setting, feedback, and building relationships based on trust and confidentiality. 


Consider other ways to integrate EE into district and school priorities, professional 
development, hiring practices, and induction and mentoring. For instance, as an example of 
alignment between an initiative and the system, one school focused on project-based learning 
drew connections to two FFT components—3c: engaging students in learning, and 2e: 
organizing physical space. 


Provide opportunities for cross-district collaboration on goal development and monitoring for 
educators in similar roles who lack within-district collaborative partners (i.e., educators in 
rural contexts or highly specialized fields). 


Leverage district-level EE evaluation reports (i.e., the EDSR survey results created by the 
office of Socially Responsible Evaluation in Education [SREed] at UW-Milwaukee) through 
participation in the EE Exchange. 
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Policy Perspective and Response by DPI EDS Director 


DPI committed to an evaluation of the EE System more than 6 years ago to inform ongoing 
improvements to the design, as well as the development of aligned training and implementation 
supports. The resulting partnerships with WCER and SREed out of the University of Wisconsin— 
Madison and the University of Wisconsin—Milwaukee, respectively, produced real-time, ongoing 
information regarding the EE System. DPI deeply values its partnership with WCER and SREed 
because of the key information it affords and the resulting impact on the design and delivery of 
the EE System. 


In the pilot stages and early years of implementation, DPI utilized these findings to make 
timely adjustments to the design of the System to meet the needs of educators and to improve the 
System’s ability to support continuous improvement in a meaningful and authentic manner. As 
changes to the System design have diminished, evaluations have shifted towards implementation 
quality and its impact on educator perceptions and practices within schools and districts. With 
this new focus, evaluation findings have primarily informed the design and development of 
resources, training, and supports to increase understanding of the EE System and improve 
implementation quality in a manner that positively influences adult practices and student 
outcomes across the state. 


The Learning-Centered Study exemplifies this valuable partnership. Study findings reinforce 
existing efforts and will inform the development of future supports. The following sections 
describe how DPI specifically intends to utilize findings from the Learning-Centered Study. 


Marketing and communications resources about the system. Many districts, including 
those described throughout the study, immediately believed in the potential of the EE System 
and worked to implement it as a continuous improvement process embedded within all aspects of 
authentic daily practice for teachers and leaders. However, many districts did not understand the 
EE System or its intended use and, as a result, implemented the system in a cursory manner to 
comply with minimal state requirements. While DPI has developed resources and supports to 
improve understanding and implementation of the EE System over the years, we understand that 
some districts may not engage with these resources (or the System) unless it has proven to be 
worth the time and effort (i.e., impact on adult practices, student achievement, and/or other 
outcomes). 


Recent evaluation efforts have focused on the quality of implementation of the EE System 
and the resulting impact (i.e., school climate and culture, educator satisfaction and retention, and 
changes in educator practice to improve student outcomes) in order to provide information to 
these districts about how educator effectiveness can be foundational to improvement efforts 
within their schools and districts. The Learning-Centered Study furthers that goal by providing 
specific examples from districts implementing with a learning-centered focus and the benefits 
they have realized as a result. With these findings, DPI will: 


e work with its internal Communications team to design and publish a series of 
informational and marketing resources to increase understanding of the EE System and 
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its positive benefits including, but not limited to, infographics, summaries, case studies, 
blogs, and podcasts; 


ask WCER and SREed to continue evaluation efforts with a focus on the relationship 
between implementation quality and student outcomes after 5 years of statewide 
implementation to continue to inform impacts of implementation; 


work with the DPI communications staff to create engaging resources intended to provide 
examples of promising practices from the Learning-Centered Study to schools and 
districts across the state; 


ask WCER and SREed to develop a similar Learning-Centered Study focused on the 
implementation of principal evaluations within the EE System across the state to further 
this learning with regards to the principal evaluation process; 


continue current efforts to create coherent training supports with statewide partners, 
including the Wisconsin Association for School District Administrators, the Association 
of Wisconsin School Administrators, and the 12 regional centers for educational services 
to ensure consistency in messaging and intended outcomes with regards to the EE 
System; 


redesign any existing resources that do not immediately increase understanding of the 
intended use of EE (e.g., a focus on the adaptive nature instead of the technical aspects of 
the System) and its role in systematic continuous improvement efforts in order to 
improve the general perception of the work in the field; and, 


continue efforts to increase understanding of the System and its intent internally and to 
work with other DPI teams to integrate EE meaningfully into their programs and 
initiatives (and vice versa) to increase opportunities for consistent messaging. Current 
cross-agency efforts include: 


oO redesigning the DPI-designed statewide data inquiry tool to explicitly connect the 
queries to the principal and teacher SLO processes, resulting in a high-quality 
SLO goal and associated educator practices/strategies; 


O writing a statewide definition of high-quality professional learning that describes 
EE as the definition of high-quality professional learning opportunities; 


O including EE as an evidence-based strategy with regards to ESSA requirements in 
order to focus local efforts on improving an existing high-quality strategy, rather 
than adding new strategies upon identification; and, 


O including EE throughout communications to ESSA and IDEA identified schools 
and districts regarding the alignment of the required continuous improvement 
processes and the existing SLO requirements, as well as the ability for Targeted 
Support and Improvement schools to write a school learning objective targeting 
the gap for which they were identified as a means to meet state requirements for 
identified schools and districts. 
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Resources to improve skills necessary for learning-centered evaluation. With regard to 
observing practice and providing feedback, initial training focused comprehensively, but 
exclusively, on evaluators’ ability to identify current levels of educator practice effectively. 
Evaluation findings (reported separately) indicate most educators agree their evaluators are 
qualified to observe them and provide consistent and accurate feedback regarding their current 
levels of practice. However, due to this focus on identifying current levels of practice, evaluators 
struggle to bridge the gap between identifying where practice is and where it should go, and 
creating a detailed and strategic plan to help educators move there. Recommendations from the 
Learning-Centered study back this finding and confirm DPI must create training that improves 
evaluators’ coaching skills. The Educator Development and Support Team will: 


work with its partners to collaboratively design training that the partners will deliver. The 
training will build skills required of all coaches (e.g., building trust), as well as skills 
specific to the evaluation observation frameworks (i.e., FFT and the Wisconsin 
Framework for Principal Leadership), such as a detailed plan to improve current levels of 
practice along the continuum of indicators within a given component that incorporates 
specific suggestions for additional training and learning to support the educator in 
successfully implementing the plan; and, 


work with the cross-agency team that developed the Wisconsin Coaching Competencies 
Profile to inform the design and delivery of coaching training. 


The evaluation of the EE System has consistently identified a need for additional training and 
supports to improve data and assessment literacy. The Learning-Centered Study confirms this 
ongoing need, specifically within the findings associated with SLO practices. In response, the 
Educator Development and Support Team will: 


continue to plan a 3-year professional learning series with national experts in 
instructional and assessment practices to support learning in these areas; 


work with the DPI Office of Educational Assessment to integrate their training resources 
on formative assessment practices into existing and new EE training opportunities to 
improve SLO practices; 


work with the DPI Office of Educational Assessment to design additional resources, as 
necessary, to improve SLO practices; 


continue to work with the DPI Title I, Special Education, Data, and Office of Educational 
Assessment Teams to revise the statewide data inquiry tool to make explicit connections 
to the teacher and principal SLO processes so that use of the tool results in the 
development of a high-quality SLO and associated practice strategies; 


increase educators’ awareness of the updated SLO training resource: “Writing Quality 
Student/School Learning Objectives (SLO)” and help partners brainstorm how to utilize 
the tool (or the content of the tool) within their own training opportunities for educators; 
and, 
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e make improvements to the SLO rubrics and guidance with feedback from DPI SLO, 
assessment, and continuous improvement experts, and from users in the field to improve 
clarity within the resources regarding how to develop and implement a rigorous and high- 
quality SLO process. 


The Learning-Centered Study confirmed that many districts, including districts focused on 
the five principles of a learning-centered evaluation, struggle to implement EE as part of a high- 
quality continuous improvement cycle based on ongoing “mini-cycles” within a larger, annual 
cycle. In response, the Educator Development and Support Team will: 


e continue to design and deliver statewide professional learning opportunities that not only 
address continuous improvement but are designed to model continuous improvement 
(e.g., Working on the Work and Leading for Learning); 


e increase coordination with the Title I and Special Education teams as they roll out 
statewide supports for continuous improvement through the regional service centers, as a 
means to assist identified schools; and, 


e design new training resources (e.g., modules and offerings through the service centers) 
that focus on the EE System as a continuous improvement process as opposed to a series 
of technical requirements, similar to the revisions made to the teacher and principal 
evaluation user guides. 


Additional resources to improve implementation of learning-centered evaluations. The 
Learning-Centered Study finds that many districts still struggle with aspects of implementing a 
learning-centered evaluation. Two years ago, DPI realized it could not expect schools or districts 
to improve implementation if they could not accurately identify their current levels of 
implementation. As a result, DPI, in partnership with WCER and SREed, developed the EE 
Exchange, which is designed to provide district, school, and teacher leaders with unique insights 
and a rich opportunity for planning and growth in EE implementation. Districts come together to 
review reports based on their local WEDSR survey data. District teams learn about how to 
interpret the data from the team of SREed researchers that developed the survey and reports. 
District teams also learn about what climate and culture factors influence effective EE 
implementation and vice versa. This process helps schools and districts identify their current 
levels of EE implementation as compared to a state average to determine areas of strength and 
areas for growth. 


Much like the EE process, identifying current levels of practice is only the first step. The 
most important step is determining a specific and effective plan to improve levels of practice. 
DPI has struggled to provide specific guidance on how to improve levels of implementation in 
the past. The Learning-Centered Study offers some suggestions to improve this process. Based 
on these recommendations, DPI will work with WCER to identify examples from the Learning- 
Centered study to develop guidance and training on how to: 


e create the systems and the time necessary to develop effective teaming, observation, and 
coaching structures; 
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e integrate EE effectively into hiring, mentoring, induction, professional development 
offerings and other HR processes; and, 


e integrate EE effectively into existing school goals and initiatives. 


DPI will support local implementation by removing the need for schools and districts to 
create some communications and resources for EE locally. Specifically, DPI will create 
customizable templates with important and necessary EE information for districts to use within 
their local systems, such as EE orientation materials. 


DPI will help focus local implementation efforts on achievement gaps and supporting the 
needs of traditionally marginalized students through revised guidance and forms. Additionally, 
DPI is considering including a requirement regarding the use of EE information to inform 
equitable practices, which would be included in all monitoring efforts. 


Finally, DPI will continue preliminary discussions with Wisconsin principal preparation 
programs to include EE training within their programs to reduce time spent with on-the-job EE 
training. This would include, but is not limited to, training in all aspects of developing and 
implementing effective SLOs and PPGs, and coaching and feedback. 


Conclusion and Future Studies 


Although the schools and districts in the study may not be representative in the state in how 
they have implemented the teacher evaluation process, the study helps to illustrate the potential 
benefits of a learning-centered evaluation approach. Study-based recommendations and policy 
responses by DPI speak to near-term change to communicate about and provide resources 
supporting learning-centered evaluation practices in Wisconsin. 


Future analysis will examine student learning trends in participating schools compared with 
schools matched for similar characteristics to address if and how implementation may impact 
student performance. Follow-up research could also identify promising EE practices for 
enhancing educator retention. Finally, the study authors plan a complementary principal 
learning-centered study to identify how districts support school leadership development through 
the principal evaluation process. 
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Appendix A: Districts 


Phase 2 Learning-Centered Districts and Schools, 2017-2018 


DPI REPORT CARD 
DISTRICT SCHOOL ENROLLMENT RATING 
Aaabeo Al Behrman 349 Exceeds Expectations 
Elementary 
Baraboo Jack Young Middle Bee Meets Expectations 
School 
Baraboo Baraboo High 969 Meets Few Expectations 
School 
casio Cashton Elementary 302 Exceeds Expectations 
Gachion (SE 00n Middle & 310 Exceeds Expectations 
High School 
Franklin Public Country Dale 502 Significantly Exceeds 
Elementary Expectations 
Hankin Public Forest Park Middle 71 Exceeds Expectations 
School 
Franklin Public Franklin High 1,512 Exceeds Expectations 
School 
ils Wren Wales Elementary 358 Exceeds Expectations 
Kettle Moraine Significantly Exceeds 
Kettle Moraine Middle School 878 Fepectarions 
Kettle Moraine Kettle Moraine High 1,021 Significantly Exceeds 
School Expectations 
Ni iwaues Oliver Wendell 784 Meets Few Expectations 
Holmes School 
Milwaukee Wedgewood Park 929 Meets Expectations 
International School 
Meals Grant Elementary 220 Exceeds Expectations 
ern John Muir Middle 1,000 Exceeds Expectations 
School 
Wausau Wausau West High 1,308 Meets Expectations 
School 


Source: DPI District Report Card, 2016-17, Summary 
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Appendix B: Study Methods 


Teacher, Evaluator, and Coach Selection 


For the non-Milwaukee districts, after receiving permission to conduct the case studies, we 
selected three core content or special education teachers per school for interviews during each of 
the case study visits. Interviews also included evaluators and coaches. A total of 46 teachers, 31 
evaluators, and 18 coaches participated in the study. 


In MPS, permission to conduct the case studies was delayed until December 2017. To 
expedite the process of recruiting participants, we asked the two MPS schools to assist in 
identifying teachers in their Summary Year during the 2017-18 school year for possible 
participation. In addition, we asked evaluators and coaches to participate. The evaluation team 
then conducted site visits during Winter and Spring 2018 at each school. 


Interview Process 


Semi-structured interviews represented the primary data collection method using protocols 
for teachers, evaluators/principals, and coaches (see Appendix C). In-person interviews occurred 
in Fall 2017 and Winter and Spring 2018. This timeframe aligned with three EE cycle milestone 
meetings: planning session, mid-interval review, and end-of-cycle conference. Interview content 
focused on EE implementation, the five learning centered principles, and teaching practice 
impacts. One to two team members conducted interviews at each site. Each participant allowed 
audio recordings of interviews, which supported extensive notes and verified quotations. 
Interviews ranged from 30 to 60 minutes in duration. Across the year, we conducted 251 
interviews. 


Document Review 


School and district documents obtained during visits from district staff, principals, coaches, 
and teachers, and from the district website, provided additional EE context. Examples include 
annual district and school strategy/improvement plans and descriptions of supporting practices 
such as professional development. 


Feedback Logs 


In addition to interviews and documents, we asked teachers to reflect on and complete logs 
on the nature of the feedback they received during the EE process, especially after announced 
and unannounced observations, mini-observations, and mid-interval and end-of-cycle 
conferences. We also asked teachers to rate (using a 1-4 scale) the quality of each feedback 
instance on its accuracy, specificity, timeliness, actionability, inclusions of strengths/weaknesses, 
relationship to the FFT, and next steps. Teachers were provided a small individual or school 
stipend to participate in the log reflections and were compensated regardless of whether they 
completed the logs. A total of 44 teachers across the six districts provided responses on about 
109 interactions. 
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SLOs and PPGs 


We obtained 653 SLO and PPG goal plans from all schools except Milwaukee. The plans 
were from 2016-2017, the year prior to the school visits. Of those, we randomly selected 10% (5 
per school), yielding a total of 81 SLOs and PPG statements for review. The SLO quality review 
focused on the seven dimensions from the DPI SLO Quality Indicator Checklist: baseline data, 
learning content, student population, targeted growth, time interval, evidence sources, and 
instructional strategies. PPG goal statements were also rated for quality. Using a four-point 
rating scale, two reviewers assessed each factor. Inter-rater agreement was acceptable, with 92% 
of the ratings within one level of agreement (perfect agreement for 45.6% of the ratings and 
46.3% agreement within one rating scale point). 


Few 2017-2018 PPGs were provided by educators during the site visits, thus prohibiting 
meaningful analysis of PPGs. Instead, we relied on descriptions of practice goals provided 
during interviews and in feedback logs for some of the goal and feedback analysis. 


Analysis and Reports 


For each school, the major analysis focused on identifying themes contained in the 
interviewees’ responses, supplemented by other supporting information. Site leads analyzed 
interviews and prepared analytic memos following each visit. At the end of the year, site leads 
prepared a tentative summary report. Another team member reviewed the report for clarity and 
understanding. We sent draft reports to school principals for review and asked them to correct 
inaccuracies and clarify perceived ambiguities. 


The school reports served as the foundation for final summary district reports. The district 
reports contained sections on basic EE implementation, consistency of EE practices with the five 
learning-centered principles, positive/promising practices, challenges, and questions for school 
and district consideration. One team member drafted each report and another team member 
reviewed the report for edits and clarifications. We sent a copy of the district report to the district 
administrator for review and feedback. A meeting occurred with a district administrator and 
other district and school leaders from each site to review and discuss the report. District 
administrators were asked to raise questions, present clarifications, or correct inaccuracies. A 
few minor clarifications were provided about district structures and supports, which were 
incorporated into the final reports. Because of the nature and generalizability of a large urban 
district, we did not draft for MPS a separate district report. Instead, school reports were sent to 
each school and to the district, with follow-up meetings held to discuss findings. 
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Appendix C: Interview Protocols 


Teacher Questions: Fall 


School context/priorities 


1. 
2: 


3. 


What are your school’s main instructional priorities? 

What are the key student equity concerns in your school? (For example, any sub-groups 
[special education, socio-economic, or race-based] that require targeted interventions.) 
How does your evaluation process help address the main priorities and equity issues? 


Learning-centered features 


4. 


How does your principal try to establish a trusting relationship with teachers around the 
educator evaluation process? 

How does your principal encourage collaboration among teachers around the educator 
evaluation process? (Probe: Collegial feedback, collaborative examination of practice.) 
How does the principal encourage use of the Danielson Framework for Teaching as a model 
of good teaching? (Probe: In conversations outside of the evaluation context, i.e., 
professional development, instructional planning.) Please provide examples. 

Can you think of examples where teachers use the Framework for Teaching as a model of 
good teaching? (Probe: In conversations about practice; during professional learning 
opportunities.) 

How does your principal encourage you to use the educator evaluation system as a process of 
continuous improvement? 


Evaluation process/support 


7; 


How does your principal integrate the educator evaluation process into other practices that 
support teachers? 


10. What supports are made available to help you get the most benefit from the evaluation 


process? (Probe: Training, coaching, dedicated time.) 


11. Do you have anything else you would like to add? 


School Administrator/Evaluator Questions: Fall 


School context/priorities 


ie 
2: 


What are your school’s main instructional priorities? 

What are the key student equity concerns in your school? [For example, any sub-groups 
(special education, socio-economic, or race-based) that require targeted interventions] 
How does your district evaluation process help address your school’s main priorities and 
equity issues? 
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Learning-centered features 


4. 


In what ways do you try to establish a trusting relationship with your teachers around the 
educator evaluation process? 

In what ways do you encourage collaboration among teachers around the educator evaluation 
process? (Probe: Collegial feedback, collaborative examination of practice.) 

To what extent do you use the Danielson Framework for Teaching as a model of good 
teaching practice in your work with teachers? (Probe: Outside the evaluation context, i.e., 
professional development, instructional planning.) Please provide examples. 

In what ways do you encourage teachers to use the educator evaluation system as a process 
of continuous improvement for their teaching practice? 

How do you integrate the educator evaluation process into other practices that support 
teachers? 


Evaluation process/support 


9. 


10. 


11. 


12. 


13. 


14. 
15. 


What educator evaluation supports are offered to the teachers in your school (i.e., training, 
coaching, dedicated time)? 

What supports are still needed in order to strengthen the evaluation process? (Probe: to 
support teacher and school priorities, including equity issues.) 

What educator evaluation supports are offered to you as an evaluator (i.e., training, coaching, 
dedicated time)? 

What evaluator supports are still needed to strengthen the evaluation process? (Probe: to 
support teacher and school priorities, including equity issues.) 

What educator evaluation supports are offered to the coaches in your school (i.e., training, 
coaching, dedicated time)? 

What coaching supports are still needed to strengthen the evaluation process? 

Do you have anything else to add? 


District/School Coach Questions: Fall 


School context/Priorities 


1. 
2. 


What are this school’s main instructional priorities? 

What are the key student equity concerns in your school? (For example, any sub-groups 
[special education, socio-economic, or race-based] that require targeted interventions.) 
How does your district evaluation process help address the school’s main priorities and 
equity issues? 


Learning-centered features 


4. 


ap 


How does the principal at this school try to establish a trusting relationship with teachers 
around the educator evaluation process? 

In what ways does the principal encourage collaboration among teachers around the educator 
evaluation process? (Probe: Collegial feedback, collaborative examination of practice.) 
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6. To what extent do you use the Danielson Framework for Teaching as a model of good 
teaching in your work with teachers here? Please provide examples. 

7. In what ways does the principal encourage teachers to use the educator evaluation system as 
a process of continuous improvement? 

8. How does the principal integrate the district evaluation process into other practices that 
support teachers? 


Evaluation process/support 


9. What educator evaluation supports are offered to the teachers in this school (i.e., training, 
coaching, dedicated time)? 

10. What teacher supports are still needed in order to strengthen the evaluation process? (Probe: 
To support teacher and school priorities, including equity issues) 

11. What educator evaluation supports are offered to the coaches (i.e., training, 
coaching/feedback, dedicated time)? 

12. What coaching supports are still needed to strengthen the evaluation process? 

13. Do you have anything else you would like to add? 


Teacher Questions: Winter 


1. What information did you receive at the start of the year about your Summary Year 
evaluation? (Probe: Was there an orientation meeting? who conducted? when did it occur?) 

2. Did you meet with your evaluator to discuss your Educator Effectiveness Plan (professional 
practice and student learning objective goals) and the evaluation process? If so, what was 
discussed? 

3. Please describe the process you went through to develop your Student Learning Objective 
(SLO) and Professional Practice Goal (PPG). 

4. How, if at all, did you use the SLO quality criteria and SLO rubric? (Probe: Do you reference 
these references at other times during the year?) 

5. How were you encouraged and supported in setting challenging goals? (Probe: From 

principal, coaches or peers?) 

Are your SLO and PPG aligned with school goals and priorities? How so? 

How, if at all, does your SLO or PPG address student equity issues? 

What, if any, supports are available to help you achieve your SLO and PPG goals? 

In what ways does your evaluator provide you with instructional feedback? (Refer to their 

feedback form [if completed] for probes.) 

10. How have you used the feedback provided by your evaluator? (Probe for why they may not 
have used the feedback.) 

11. Have you received any feedback from your school’s coach or peers? Please provide 
examples of the feedback you received and how you used the feedback. (Probe: Related to 
student equity goals or issues.) 

12. Have you had your mid-year review? If so, when did it occur? If not, is the date set? 

13. Do you have anything else you would like to add? 


OOO SD 
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Evaluator Questions: Winter 


1. 


© 


10. 


11. 
12. 


What information did teachers receive at the start of the year about their Summary Year 
evaluation? (Probe: Was there school or district orientation meeting? Who conducted? 
when?) 

How did you help teachers develop their Student Learning Objectives (SLOs)? 

How, if at all, do you use the SLO quality criteria and rubric to help teachers with their 
SLOs? (Probe for example.) 

How did you work with teachers on their Professional Practice Goals (PPGs)? 

How do you encourage your teachers to set challenging or “stretch” goals? 

What evidence do you have that teachers are setting challenging goals? 

How do you try to align teachers’ SLOs and PPGs to school goals and priorities? (Probe 
about school SLO.) 

In what ways do you support teachers in trying to achieve their goals? 

In what ways do you try to give specific instructional feedback to your teachers? (Probe: 
Related to student equity goals or issues.) 

I’d like to hear about example of feedback you may have provided to the teachers in this 
study (teacher 1 , teacher 2__, teacher 3 ). Can you talk about examples for 
each teacher? 

How do you know teachers are using the feedback you provide? 

Do you have anything else you would like to add? 


District/School Coach Protocol: Winter 


dhs 


i 


11. 
Le 


Please tell me about the types of coaching activities you have participated in with teachers 
since we last spoke. 

Have you worked with the teacher’s (insert names) in the Learning-Centered Study? If so, 
please reference those interactions when answering the following questions. 

How do you support teachers’ efforts to align SLOs and PPGs to school goals and priorities? 
How do you support teachers’ efforts to structure SLO’s and PPG’s to address student equity 
issues? 

In what ways do you give specific instructional feedback to your teachers? (Probe: frequency 
of these activities. ) 

How do you work with teachers on their SLO and PPG goals? 

How, if at all, do you use the SLO quality criteria and SLO rubric in your work with 
teachers? 

How do you encourage teachers to set challenging or “stretch” goals? 

How do you know teachers are setting challenging goals? Can you give an example? 


. Do you have the opportunity to follow up with teachers after you provide them feedback? 


Can you share examples? 

To what extent do your conversations with teachers address student equity goals or issues? 
Do you have anything else you would like to add? (Probe: How else do you support teachers 
in the school [beyond EE]?) 
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Teacher Questions: Spring 


iL: 


10. 


11. 


12. 


How has the SLO and PPG process impacted your practice? (Probe for changes to 
instruction, assessment, collegial interactions.) 

How often were you observed? Did you receive feedback after each observation? 

How have you applied instructional feedback from your evaluator? Have you used the 
feedback for setting personal goals for improvement? 

Add follow-up probe for specifics related to feedback log responses. 

Looking back over the year, how has your local EE process involved collaboration (with 
peers, with coaches, with evaluators)? 

What other new or revised teaching practices are you implementing? How have they helped 
your practice? How has the evaluation process supported these practices? 

Have any of these changes you have discussed related to your school’s improvement 
priorities? Please elaborate. (Probe for any impact on school or classroom equity priorities.) 
How are your SLOs scored? (Probe for self-scoring and administrator scoring in summary 
and supporting years.) 

Have you had your end-of-cycle conference yet? What do you hope to gain from the end-of- 
cycle conversation? 

Based on the evaluation process you went through this year, how are you going to use the 
information to inform your goals for next year? 

Reflecting on the year, what additional supports would have helped you get the most benefit 
from the evaluation process? 

Do you have anything else you would like to add? 


Evaluator Questions: Spring 


1. 


Thinking about the overall evaluation process, what new or revised teaching practices have 


you observed in teacher __, , and ? (Probe for changes related to SLO and PPG 
process.) 
Since we last spoke, can you summarize your feedback to teachers__,__—, and__ about 


their practice? (Probe for examples.) 

How have the teachers applied your feedback? (Probe for whether it led teachers to change 
practices to improve instruction.) 

How many other teachers do you evaluate this year? (Confirm if you already know.) 

How has the evaluation process contributed to addressing your school’s improvement 
priorities? 

Have any of these changes related to the equity concerns in your school? Please explain. 
Can you describe the process you use to score teacher SLOs (i.e., use SLOs from all three 
years or just the Summary Year; how teacher self-scores are considered)? 

Have you scheduled your end-of-cycle conferences yet? What are your goals for these 
conversations? 

Reflecting on the year, are you going to make any changes to your evaluation approach next 
year? (Probe: What is the rationale for any changes?) 
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10. What would you change about the local evaluation process to increase the potential to 


improve teaching? 


11. In general, are there any supports that would help you improve your leadership practice? 


Would any of those be especially helpful in improving the EE process? 


12. Do you have anything else you would like to add? 


District/School Coach Questions: Spring 


1. 


Since we last spoke, can you summarize your coaching activities with teacher__,__, and 

___? (Probe for examples and feedback specific to their SLOs and PPGs.) 

If your work with these teachers has involved feedback about their practice, can you talk 

about examples? 

a. How have teachers applied your feedback to their practice? (Probe for nature and depth 
of changes.) 

How helpful do you believe teachers in this school find the overall evaluation process? 

(Probe for SLO and PPG process, feedback.) 

What do you believe would improve the evaluation process (including the SLO and PPG 

process)? 

Are you aware of new or revised teaching practices that have come about from the overall 

evaluation process? Can you provide examples? 

How has the evaluation process contributed to addressing your school’s improvement 

priorities? 

Have any of these changes related to the equity concerns in your school? 

Are there any forms you use to support your coaching that we could collect? (If you haven’t 

already.) 

Are there any changes to your coaching role that would better support teachers in your 

school? 


10. Do you have anything else you would like to add? 
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